A typical multiprocessor computer system comprises several processors, cache memories of different hierarchy levels, storage controlling devices, a main memory, and means that control data exchange with I/O devices.
When describing the different memory levels the processors address, one has to start with the first level cache. Each processor has its own, individually assigned first level cache integrated on-chip. Therefore, the wiring delay is small for read and write operations to this cache and the latency is low, which means that a very fast access to the data and instructions stored in the first level cache is possible.
The next level in the hierarchy of cache storage means is the second level cache. The task of this cache is to provide data exchange with the first level cache. The storage size of the second level cache is much greater than that of the first level cache, and usually, the second level cache is not integrated on the processor chip, but placed on a dedicated chip (or set of chips). Therefore, the latency for a second level cache access is not as low as for the first level cache.
There exist several concepts for the realization of a second level cache structure in a multiprocessor computer system.
One of these concepts is to assign an individual second level cache to each processor, which means that each processor can only address the data in its personal cache. Therefore, each of these "dedicated caches" holds exactly the data units requested by its processor; and of course, two of these caches usually hold different data units.
Let us suppose that processor 1 writes to a cache line contained in its second level cache, and that a copy of said line is also contained in a second level cache owned by a different processor, e.g. by processor 3. In the moment the write access to cache 1 occurs, the page in cache 3 is not valid any more. Therefore, a mechanism has to exist (the so-called switching-and-snooping network) that signals to cache 3 that the data of a certain line has become invalid, in order to maintain data integrity.
The necessity of keeping track of different versions of one and the same cache line in the second level cache falls away when a shared cache structure is realized. "Shared" means that each processor addresses the same second level cache storage space. The accessible lines that have been copied to and that are now contained in the second level cache are the same for all processors.
This also implies that a shared second level cache built in order to have a performance similar to that of individually assigned second level caches has to have an absolute storage size that is several times the size of one of said individual caches. When realizing said shared cache concept, one usually employs several second level cache chips in parallel.
Between second level cache and processors, data is transferred via so-called processor busses. As a certain line that a processor wants to address can be stored on any of the second level cache chips, each second level cache chip has to be connected to each of the processors of the multiprocessor system via one of said processor busses. In order to take advantage of a low latency of the second level cache, each processor bus has to be capable of high-speed data transmission and provide a high-bandwidth communication link. Usually, regular instruction and data flow on the processor bus, from the second level cache to the processor, and vice versa, only requires about 20% of the maximum data transfer rate possible on said processor bus.
All the devices described so far, the processors and the second level cache chips, may be mounted on one multi-chip module. This module exchanges data via memory busses with a main memory, consisting of at least one memory card. Each memory card comprises a storage controller, which also provides the clocking system for the whole memory card, and several memory banks.
When a processor requests data which is not accessible in the second level cache yet, lines have to be transferred from a memory bank of the main memory to the second level cache. On the other hand, cache lines that have been modified in the second level cache have to be rewritten to the main memory in order to ensure data integrity.
Data exchanged between the computer system on the one hand and external devices, such as discs, tapes, etc. on the other hand is generally referred to as I/O-traffic. Usually, I/O devices are coupled to an I/O adapter, which is connected to the computer system via the second level cache. One or two of the processors of the multiprocessor system, the so-called service assist processors (SAPs), are especially responsible for handling data exchange between the I/O devices and the main memory.
In a multiprocessor system, each of the second level cache chips has to accomodate processor busses connecting said second level cache chip to each of the processors, memory busses linking said cache chip to a memory card, and additional data links for exchanging data with I/O devices. All these connections require a lot of the second level cache chip's pins.
Usually, a chip has a side length of 14.5 or 16 mm. The lower side of the chip bears an array of blobs of solder, in order to connect the chip to the ceramic substrate, the multi-chip module. For a typical chip, the number of soldering tags is 1240. One would be tempted to increase the number of pins by simply increasing the chip size. When soldering the chip to the substrate according to the "flip chip technique" (described, for example, in L. F. Miller's article "Controlled Collapse Reflow Chip Joining", IBM J. Res. Develop., May 1969, p. 239-250), und thus heating up both the chip and the substrate, shear strains arise because of the different coefficients of expansion of chip and substrate. The occurence of these shear strains imposes physical limits to the increase in chip size and to the increase in pin count.
Therefore, pins are valuable, and any method that contributes to saving pins, especially pins of a second level cache chip, permits a better layout and improved performance.
One method for saving pins is the introduction of switching means. For example, both a memory bus and a data link to an I/O adapter can be coupled to a switching network which, for its part, is connected to a second level cache chip. Thus, the memory bus and the I/O data link share the same cache chip pins, and therefore, a lot of the pins that formerly were necessary to accomodate both the memory bus and the I/O data link become available. Of course, one does not have to switch between a memory bus and an I/O data link, one could as well switch between links to different I/O devices, or, in case a multiprocessor system with a clustered shared second level cache is realized, between connections to different clusters.
The introduction of switches has several disadvantages, though. For example, let us consider the case that both an I/O data link and a memory bus are connected via a switch to a second level cache chip. When data traffic occurs on both the I/O data link and the memory bus, the switch can only permit access to the second level cache to one of them, which means that the other one can't exchange data with the cache at that time and has to wait for several cycles. Furtheron, the implementation of additional switches requires additional chips. Increasing the number of chips always means that packaging becomes more difficult and that the wiring delay of certain signals increases. The main disadvantage of the use of switches is therefore the occurence of additional delays which severely degrade system performance.
So far, reference has only been made to a single multiprocessor system. In order to further increase system performance, concepts of parallel computing have been realized where several of said multiprocessor systems are connected. Each of these multiprocessor systems still addresses its individual memory, but there also exists a central shared memory which every multiprocessor system can address. The storage size of this central memory is much larger than the size of the memories individually assigned to each multiprocessor system. For each data structure (list, cache, lock structure) contained in said central memory, ownerships are defined which enable or disable access of a certain multiprocessor system to said data structure. One of the multiprocessor systems, the so-called coupling facility, is dedicated to managing both the communication/synchronization between the different multiprocessor systems and the access (and access rights) of the multiprocessor systems to the central memory which is attached to said coupling facility. This implies that all data that one of the multiprocessor systems fetches from or stores to the central memory has to be exchanged between said multiprocessor system and the coupling facility. Thus, strong data links have to exist between the systems and the coupling facility, and the bandwidth required for these links in order to permit quick access of all multiprocessor systems to the central memory is much higher than the bandwidth required for the transmission of conventional I/O traffic between a computer system and its external devices. Therefore, with respect to a coupling facility, the question arises of how to connect such an additional intersystem communication link to the computer system.
Again, one solution is the introduction of switching means in order to couple said intersystem communication link to the second level cache. With this method, it is possible to accomodate the link and still meet pin count requirements. Because of the additional wait cycles caused by switching means, the latency of an access to the computer system is increased. Thus, the introduction of switching means might be a correct concept for a coupling facility of today, but for future parallel systems, latency will have to be much lower.